Solving POMDP by On-Policy Linear Approximate Learning Algorithm

This paper presents a fast Reinforcement Learning (RL) algorithm to solve Partially Observable Markov Decision Processes (POMDP) problem. The proposed algorithm is devised to provide a policy-making framework for Network Management Systems (NMS) which is in essence an engineering application without an exact model. The algorithm consists of two phases. Firstly, the model is estimated and policy is learned in a completely observable simulator. Secondly, the estimated model is brought into the partially observed real-world where the learned policy is then ne-tuned. The learning algorithm is based on the on-policy linear gradient-descent learning algorithm with eligibility traces. This implies that the Q-value on belief space is linearly approximated by the Q-value at vertex over the belief space where on-line TD method will be applied. The proposed algorithm is tested against the exact solutions to extensive small/middle-size benchmark examples from POMDP literature and found near optimal in terms of average-discounted-reward and step-togoal. The proposed algorithm signi cantly reduces the convergence time and can easily be adapted to large state-number problems.